A Novel Measure of Surprise with Applications for Learning within Changing Environments

Surprise is informative because it drives attention [1] and modifies learning [2]. Correlates of surprise have been observed at different stages of neural processing, and found to be relevant for learning and memory formation [3]. Although surprise is ubiquitous, there is neither a widely accepted theory that quantitatively links surprise to observed behavior, such as the startle response, nor an agreement on how surprise should influence learning speed or other parameters in iterative statistical learning algorithms. Building on and going beyond earlier surprise measures [1, 4–6], we propose a novel information theoretic measure for calculating surprise in a Bayesian framework so as to capture uncertainty of the world as well as imperfections of the subjective model of the world, two important aspects of surprise. The principle of future surprise minimization leads to a learning rule that can be interpreted as a surprise modulated belief update suitable for learning within changing environments. Importantly, we do not need an assumption about how quickly the world changes. We apply our surprise-modulated learning rule to an exploration task in a maze-like environment and to a dynamic decision making task. Our results are consistent with the behavioral finding that surprising events induce humans and animals to learn faster and to adapt more quickly to changing environments [7–9].